Papers with visual alignment
Expressing Visual Relationships via Language (P19-1)
Copied to clipboard
| Challenge: | Current studies on image captioning focus on single image, but there are no effective models for generating relational captions for two images. |
| Approach: | They propose a language-guided image editing dataset that contains real image pairs with corresponding editing instructions. |
| Outcome: | The proposed model outperforms baseline and existing methods on two datasets. |
VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation (2026.acl-long)
Copied to clipboard
Yixia Li, Yaqing Shi, Zhiwen Ruan, Dongdong Zhang, Lingjie Jiang, Shaohan Huang, Yun Chen, Guanhua Chen, Furu Wei
| Challenge: | Multimodal large language models have advanced rapidly, yet most remain English-centric . scaling multilingual multimodal instruction tuning is limited by the scarcity and high cost of non-English image–text supervision. |
| Approach: | They propose a framework that decouples multilingual language enhancement from visual alignment by composing complementary task vectors over a shared LLM backbone. |
| Outcome: | The proposed framework achieves competitive performance with a fully multimodally trained model using less than 2% of the text data. |
iVISPAR — An Interactive Visual-Spatial Reasoning Benchmark for VLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) struggle with spatial reasoning and visual alignment, despite their performance on 2D tasks. |
| Approach: | They propose a multimodal benchmark to evaluate VLMs' spatial reasoning capabilities based on the sliding tile puzzle . |
| Outcome: | The proposed model performs better on 2D tasks compared to 3D or text-based settings, but struggles with complex spatial configurations and consistently falls short of human performance. |